{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "ce0b0e62",
   "metadata": {},
   "source": [
    "\n",
    "**修改部分**：\n",
    "1. 全局设置：pandas 一些安全选项、matplotlib 更清晰输出。\n",
    "2. 自动为 `pd.read_csv` 补 `low_memory=False`（若未指定）。\n",
    "3. 在 Notebook 末尾添加“自动数据优化”单元：运行时会对所有 DataFrame 做 parse_dates、设索引、类别压缩与数值 downcast，并显示内存变化。\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "85504adb",
   "metadata": {},
   "outputs": [],
   "source": [
    "# === 全局配置（优化项：不改变计算逻辑） ===\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "from pathlib import Path\n",
    "\n",
    "# 尽量减少链式赋值副作用（pandas>=2.0）\n",
    "try:\n",
    "    pd.options.mode.copy_on_write = True\n",
    "except Exception:\n",
    "    pass\n",
    "\n",
    "# 更清晰的绘图输出\n",
    "plt.rcParams['figure.dpi'] = 140\n",
    "plt.rcParams['savefig.bbox'] = 'tight'\n",
    "\n",
    "# 其他方便的全局变量\n",
    "DATA_DIR = Path('data')\n",
    "DATA_DIR.mkdir(exist_ok=True)\n",
    "print('全局配置已应用')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "54fb82cc",
   "metadata": {},
   "outputs": [],
   "source": [
    "# === 单元格 1: 导入依赖库 ===\n",
    "import numpy as np # 数据处理最重要的模块\n",
    "import pandas as pd # 数据处理最重要的模块\n",
    "import scipy.stats as stats # 统计模块\n",
    "import scipy\n",
    "# import pymysql  # 导入数据库模块\n",
    "\n",
    "from datetime import datetime # 时间模块\n",
    "import statsmodels.formula.api as smf  # OLS regression\n",
    "\n",
    "# import pyreadr # read RDS file\n",
    "\n",
    "from matplotlib import style\n",
    "import matplotlib.pyplot as plt  # 画图模块\n",
    "import matplotlib.dates as mdates\n",
    "\n",
    "from matplotlib.font_manager import FontProperties # 作图中文\n",
    "from pylab import mpl\n",
    "#mpl.rcParams['font.sans-serif'] = ['SimHei']\n",
    "#plt.rcParams['font.family'] = 'Times New Roman'\n",
    "\n",
    "#输出矢量图 渲染矢量图\n",
    "%matplotlib inline\n",
    "%config InlineBackend.figure_format = 'svg'\n",
    "\n",
    "from IPython.core.interactiveshell import InteractiveShell # jupyter运行输出的模块\n",
    "#显示每一个运行结果\n",
    "InteractiveShell.ast_node_interactivity = 'all'\n",
    "\n",
    "#设置行不限制数量\n",
    "#pd.set_option('display.max_rows',None)\n",
    "\n",
    "#设置列不限制数量\n",
    "pd.set_option('display.max_columns', None)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6430571e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# === 单元格 2: 读取/下载数据 ===\n",
    "data = pd.read_csv('datasets/000001.csv', low_memory=False)\n",
    "data['Day'] = pd.to_datetime(data['Day'],format='%Y/%m/%d')\n",
    "data.set_index('Day', inplace = True)\n",
    "data.sort_values(by = ['Day'],axis=0, ascending=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6df9c798",
   "metadata": {},
   "outputs": [],
   "source": [
    "# === 单元格 3: 通用计算/执行 ===\n",
    "data_new = data['1995-01':'2024-09'].copy()\n",
    "data_new['Close'] = pd.to_numeric(data_new['Close'])\n",
    "data_new['Preclose'] = pd.to_numeric(data_new['Preclose'])\n",
    "# 计算000001上证指数日收益率 两种：\n",
    "data_new['Raw_return'] = data_new['Close'] / data_new['Preclose'] - 1\n",
    "data_new"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "de5ca902",
   "metadata": {},
   "outputs": [],
   "source": [
    "# === 单元格 4: 数据清洗/转换 ===\n",
    "Month_data = data_new.resample('ME')['Raw_return'].apply(lambda x: (1+x).prod() - 1).to_frame()\n",
    "Month_data.reset_index(inplace = True)\n",
    "Month_data.rename(columns = {'Day':'month'}, inplace = True)\n",
    "Month_data.set_index('month', inplace = True)\n",
    "Month_data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a4cde4f8",
   "metadata": {},
   "outputs": [],
   "source": [
    "# === 单元格 5: 数据清洗/转换 ===\n",
    "Quarter_data = data_new.resample('QE')['Raw_return'].apply(lambda x: (1+x).prod() - 1).to_frame()\n",
    "Quarter_data.reset_index(inplace = True)\n",
    "Quarter_data.rename(columns = {'Day':'Q'}, inplace = True)\n",
    "Quarter_data.set_index('Q', inplace = True)\n",
    "Quarter_data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c5cfb2f6",
   "metadata": {},
   "outputs": [],
   "source": [
    "# === 单元格 6: 数据清洗/转换 ===\n",
    "Year_data = data_new.resample('YE')['Raw_return'].apply(lambda x: (1+x).prod() - 1).to_frame()\n",
    "Year_data.reset_index(inplace = True)\n",
    "Year_data.rename(columns = {'Day':'Year'}, inplace = True)\n",
    "Year_data.set_index('Year', inplace = True)\n",
    "Year_data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e253db0b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# === 单元格 7: 读取/下载数据 ===\n",
    "inflation = pd.read_csv('datasets/inflation.csv', low_memory=False)\n",
    "inflation['month'] = pd.to_datetime(inflation['month'],format='%Y/%m/%d')\n",
    "inflation.set_index('month',inplace=True)\n",
    "inflation.sort_values(by=['month'],axis=0,ascending=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "696e7ea7",
   "metadata": {},
   "source": [
    "# 月度数据的预测\n",
    "\n",
    "A simple linear regression of an asset return on one or a few lagged predictors of interest is the most popular econometric approach for testing for return predictability. For simplicity, consider a univariate predictive regression of the period- $(t + 1)$ stock market return $r_{t+1}$ on a single predictor variable $x_t$:\n",
    "$$\n",
    "r_{t+1}=\\alpha+\\beta x_{t}+\\varepsilon_{t+1}\n",
    "$$\n",
    "where $\\varepsilon_{t+1}$ is a zero-mean, unpredictable disturbance term. When $x_t$ is the inflation rate, dividend yield, book-to-price ratio, or turnover. Many researchers find that $\\beta$ is significantly different from zero; that is, there is in-sample evidence of stock market return predictability.\n",
    "\n",
    "* H0:$\\beta = 0$\n",
    "* H1:$\\beta \\ne 0$(我们需要通过理论分析，得出$\\beta$的符号)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9a81c0a6",
   "metadata": {},
   "source": [
    "## ICAPM 模型的简化版\n",
    "\n",
    "$$\n",
    "r_{t} = \\gamma E_{t-1}(\\sigma^2_{t})\n",
    "$$\n",
    "\n",
    "$$\n",
    "r_{t} = \\gamma E_{t-1}(cov(r,\\Delta c))\n",
    "$$\n",
    "\n",
    "$\\gamma$就是相对风险厌恶系数。\n",
    "这里涉及到一个概念就是 **股权溢价之谜**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5ab2d6e4",
   "metadata": {},
   "outputs": [],
   "source": [
    "# === 单元格 10: 数据清洗/转换 ===\n",
    "market_variance = data_new.resample('ME').apply({\n",
    "    'Raw_return':\n",
    "    lambda x: sum(x**2)\n",
    "})\n",
    "market_variance.reset_index(inplace=True)\n",
    "market_variance.rename(columns={'Day':'month','Raw_return':'RV'},inplace=True)\n",
    "market_variance.set_index('month',inplace=True)\n",
    "market_variance\n",
    "\n",
    "# market_variance <- daily_data[,.(MV = sum(Raw_return^2)),by = 'month'] This is R code"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "29a79fdf",
   "metadata": {},
   "outputs": [],
   "source": [
    "# === 单元格 11: 数据清洗/转换 ===\n",
    "reg_data = pd.merge(Month_data,market_variance,on = 'month')\n",
    "reg_data = pd.merge(reg_data,inflation,on = 'month')\n",
    "reg_data\n",
    "# Output reg_data to reg_data.csv\n",
    "reg_data.to_csv('datasets/reg_data.csv')\n",
    "# Output reg_data to reg_data.xlsx\n",
    "reg_data.to_excel('datasets/reg_data.xlsx')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f0bac3cc",
   "metadata": {},
   "source": [
    "## 作图 Plot"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "919946cc",
   "metadata": {},
   "outputs": [],
   "source": [
    "# === 单元格 13: 绘图/可视化 ===\n",
    "reg_data_plot = reg_data['2000-01':'2024-09'].copy()\n",
    "# Plot the China's stock market return and inflation into one graph\n",
    "fig, ax1 = plt.subplots(figsize=(20,5))\n",
    "# the linewidth and marker size are set to be very small\n",
    "ax1.plot(reg_data_plot['Raw_return'],color='red',marker='o',linewidth=0.8,\n",
    "         markersize=4,\n",
    "         linestyle='-',label='China Stock Market Return')\n",
    "ax1.set_ylabel('China Stock Market Return',color='red')\n",
    "#ax1.set_xlabel('Month')\n",
    "\n",
    "# 设置x轴的日期显示格式\n",
    "data_format = mdates.DateFormatter('%Y')\n",
    "ax1.xaxis.set_major_formatter(data_format)\n",
    "ax1.xaxis.set_major_locator(mdates.YearLocator())\n",
    "\n",
    "# 转置x轴的日期显示格式\n",
    "plt.xticks(rotation = 90)\n",
    "\n",
    "\n",
    "ax2 = ax1.twinx()\n",
    "ax2.plot(reg_data_plot['RV'].shift(1),color='blue',marker='o',linewidth=0.8,\n",
    "         markersize=4,\n",
    "         linestyle='-',label='Realized Variance')\n",
    "\n",
    "ax2.set_ylabel('Realized Variance',color='blue')\n",
    "\n",
    "plt.title('China Stock Market Return and Realized Variance')\n",
    "\n",
    "# change the legend into one box\n",
    "lines, labels = ax1.get_legend_handles_labels()\n",
    "lines2, labels2 = ax2.get_legend_handles_labels()\n",
    "ax2.legend(lines + lines2, labels + labels2, loc='upper right')\n",
    "\n",
    "plt.show();"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b956bfbf",
   "metadata": {},
   "source": [
    "## 描述性统计 Summary\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b2002d51",
   "metadata": {},
   "outputs": [],
   "source": [
    "# === 单元格 15: 通用计算/执行 ===\n",
    "reg_data['RV'].describe().round(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "54db4d10",
   "metadata": {},
   "outputs": [],
   "source": [
    "# === 单元格 16: 通用计算/执行 ===\n",
    "reg_data['RV'].skew()\n",
    "reg_data['RV'].kurt()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bfef6dd3",
   "metadata": {},
   "source": [
    "## OLS 回归结果"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9cb3291c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# === 单元格 18: 通用计算/执行 ===\n",
    "reg_data['lRV'] = reg_data['RV'].shift(1)\n",
    "model_cpi = smf.ols('Raw_return ~ lRV',\n",
    "                 data=reg_data['1995-01':'2024-09']).fit(\n",
    "                     cov_type='HAC', cov_kwds={'maxlags': 6})\n",
    "print(model_cpi.summary())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "620f5c6a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# === 单元格 19: 通用计算/执行 ===\n",
    "reg_data['lcpi'] = reg_data['cpi'].shift(2)\n",
    "model_cpi = smf.ols('Raw_return ~ lcpi',\n",
    "                 data=reg_data['2000-01':'2024-09']).fit(\n",
    "                     cov_type='HAC', cov_kwds={'maxlags': 6})\n",
    "print(model_cpi.summary())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e7ef53b9",
   "metadata": {},
   "outputs": [],
   "source": [
    "# === 单元格 20: 通用计算/执行 ===\n",
    "model_twovariables = smf.ols('Raw_return ~ lRV + lcpi',\n",
    "                 data=reg_data['2000-01':'2024-09']).fit(\n",
    "                     cov_type='HAC', cov_kwds={'maxlags': 6})\n",
    "print(model_twovariables.summary())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "68b4173d",
   "metadata": {},
   "source": [
    "## 季度结果"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "130eb3e2",
   "metadata": {},
   "outputs": [],
   "source": [
    "# === 单元格 22: 数据清洗/转换 ===\n",
    "Q_reg_data = reg_data['1995-01':'2024-09'].resample('QE').apply({\n",
    "    'Raw_return':\n",
    "    lambda x: np.prod(1+x) - 1,\n",
    "    'RV':\n",
    "    lambda x: sum(x)\n",
    "})\n",
    "Q_reg_data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fd987467",
   "metadata": {},
   "outputs": [],
   "source": [
    "# === 单元格 23: 绘图/可视化 ===\n",
    "# Plot the China's stock market return and inflation into one graph\n",
    "fig, ax1 = plt.subplots(figsize=(10,5))\n",
    "# the linewidth and marker size are set to be very small\n",
    "ax1.plot(Q_reg_data['Raw_return'],color='red',marker='o',linewidth=0.8,\n",
    "         markersize=4,\n",
    "         linestyle='-',label='China Stock Market Return')\n",
    "ax1.set_ylabel('China Stock Market Return',color='red')\n",
    "#ax1.set_xlabel('Month')\n",
    "\n",
    "# 设置x轴的日期显示格式\n",
    "data_format = mdates.DateFormatter('%Y')\n",
    "ax1.xaxis.set_major_formatter(data_format)\n",
    "ax1.xaxis.set_major_locator(mdates.YearLocator())\n",
    "\n",
    "# 转置x轴的日期显示格式\n",
    "plt.xticks(rotation = 90)\n",
    "\n",
    "ax2 = ax1.twinx()\n",
    "ax2.plot(Q_reg_data['RV'].shift(1),color='blue',marker='o',linewidth=0.8,\n",
    "         markersize=4,\n",
    "         linestyle='-',label='Realized Variance')\n",
    "\n",
    "ax2.set_ylabel('Realized Variance',color='blue')\n",
    "\n",
    "plt.title('China Stock Market Return and Realized Variance')\n",
    "\n",
    "# change the legend into one box\n",
    "lines, labels = ax1.get_legend_handles_labels()\n",
    "lines2, labels2 = ax2.get_legend_handles_labels()\n",
    "ax2.legend(lines + lines2, labels + labels2, loc='upper right')\n",
    "\n",
    "# save figure\n",
    "fig.savefig('images/China Stock Market Return and Realized Variance Quarter.png',dpi = 1000,bbox_inches='tight')\n",
    "\n",
    "plt.show();"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ff07a774",
   "metadata": {},
   "outputs": [],
   "source": [
    "# === 单元格 24: 通用计算/执行 ===\n",
    "Q_reg_data['lRV'] = Q_reg_data['RV'].shift(1)\n",
    "model_qcpi = smf.ols('Raw_return ~ lRV',\n",
    "                 data=Q_reg_data['2000-01':'2024-09']).fit(\n",
    "                     cov_type='HAC', cov_kwds={'maxlags': 6})\n",
    "print(model_qcpi.summary())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "62c5fba9",
   "metadata": {},
   "source": [
    "## 长期预测 Long Horizon Forecast"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2098b7ea",
   "metadata": {},
   "source": [
    "$$\n",
    "r_{t+1} + r_{t+2} + r_{t+3}  =\\alpha+\\beta x_{t}+\\varepsilon_{t+1}\n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e7c0ef67",
   "metadata": {},
   "outputs": [],
   "source": [
    "# === 单元格 27: 通用计算/执行 ===\n",
    "reg_data['next_ret'] = reg_data['Raw_return'].shift(-1) + 1\n",
    "reg_data['next_ret2'] = reg_data['Raw_return'].shift(-2) + 1\n",
    "reg_data['next_ret3'] = reg_data['Raw_return'].shift(-3) + 1\n",
    "reg_data['future_3month_return'] = reg_data['next_ret'] * reg_data['next_ret2'] * reg_data['next_ret3'] - 1\n",
    "reg_data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "31f0ab37",
   "metadata": {},
   "outputs": [],
   "source": [
    "# === 单元格 28: 通用计算/执行 ===\n",
    "model_cpi_3month = smf.ols('future_3month_return ~ RV',\n",
    "                 data=reg_data['2000-01':'2024-09']).fit(\n",
    "                     cov_type='HAC', cov_kwds={'maxlags': 6})\n",
    "print(model_cpi_3month.summary())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3325ec06",
   "metadata": {},
   "source": [
    "# 预测波动率"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "95e20535",
   "metadata": {},
   "source": [
    "这里的结果显示出波动率具有非常高的自相关性，类比前一次课的通货膨胀。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d39b8499",
   "metadata": {},
   "outputs": [],
   "source": [
    "# === 单元格 31: 通用计算/执行 ===\n",
    "model_mv = smf.ols('RV ~ lRV + lcpi',\n",
    "                 data=reg_data['2000-01':'2024-09']).fit(\n",
    "                     cov_type='HAC', cov_kwds={'maxlags': 6})\n",
    "print(model_mv.summary())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "55353c76",
   "metadata": {},
   "outputs": [],
   "source": [
    "# === 自动数据优化（运行此单元来对内存中的 DataFrame 做优化） ===\n",
    "# 说明：此单元将会扫描当前 Notebook 的全局变量，寻找 pandas.DataFrame 对象，\n",
    "# 对每个 DataFrame：\n",
    "#  1) 如果存在 'Date' 或 'date' 列，会尝试解析为 datetime 并设为索引（不会删除原列）\n",
    "#  2) 将 object 列基数较低（默认<50）的转换为 category\n",
    "#  3) 对整数与浮点列做 downcast（节省内存），并打印优化前后的内存用量\n",
    "# 在完成数据读取与初步清洗后运行此单元。\n",
    "\n",
    "import gc, sys\n",
    "from pandas.api.types import is_object_dtype, is_integer_dtype, is_float_dtype\n",
    "import pandas as pd\n",
    "\n",
    "def df_memory_mb(df):\n",
    "    return df.memory_usage(deep=True).sum() / 1024**2\n",
    "\n",
    "optimized_reports = []\n",
    "g = globals()\n",
    "for name, val in list(g.items()):\n",
    "    if isinstance(val, pd.DataFrame):\n",
    "        df = val\n",
    "        before_mb = df_memory_mb(df)\n",
    "        # 1) Try parse Date column\n",
    "        for col in list(df.columns):\n",
    "            if col.lower() == 'date' and not pd.api.types.is_datetime64_any_dtype(df[col]):\n",
    "                try:\n",
    "                    df[col] = pd.to_datetime(df[col], errors='coerce')\n",
    "                except Exception:\n",
    "                    pass\n",
    "        # If there is a datetime-like column named Date, set as index (but keep column)\n",
    "        if 'Date' in df.columns or 'date' in df.columns:\n",
    "            date_col = 'Date' if 'Date' in df.columns else 'date'\n",
    "            if not pd.api.types.is_datetime64_any_dtype(df[date_col]):\n",
    "                try:\n",
    "                    df[date_col] = pd.to_datetime(df[date_col], errors='coerce')\n",
    "                except Exception:\n",
    "                    pass\n",
    "            try:\n",
    "                df = df.set_index(pd.DatetimeIndex(df[date_col]))\n",
    "            except Exception:\n",
    "                pass\n",
    "        # 2) object -> category if low cardinality\n",
    "        for col in df.select_dtypes(include=['object']).columns:\n",
    "            try:\n",
    "                if df[col].nunique(dropna=True) < 50:\n",
    "                    df[col] = df[col].astype('category')\n",
    "            except Exception:\n",
    "                pass\n",
    "        # 3) downcast numeric\n",
    "        for col in df.select_dtypes(include=['int64']).columns:\n",
    "            try:\n",
    "                df[col] = pd.to_numeric(df[col], downcast='integer')\n",
    "            except Exception:\n",
    "                pass\n",
    "        for col in df.select_dtypes(include=['float64']).columns:\n",
    "            try:\n",
    "                df[col] = pd.to_numeric(df[col], downcast='float')\n",
    "            except Exception:\n",
    "                pass\n",
    "        after_mb = df_memory_mb(df)\n",
    "        # assign optimized df back to globals\n",
    "        g[name] = df.copy()\n",
    "        optimized_reports.append((name, round(before_mb,4), round(after_mb,4)))\n",
    "        gc.collect()\n",
    "\n",
    "# Print report\n",
    "if optimized_reports:\n",
    "    print('优化报告（DataFrame 名, 优化前 MB, 优化后 MB）：')\n",
    "    for r in optimized_reports:\n",
    "        print(r)\n",
    "else:\n",
    "    print('未检测到可优化的 pandas.DataFrame，请先运行数据读取单元并确保 DataFrame 存在于全局变量。')"
   ]
  }
 ],
 "metadata": {
  "language_info": {
   "name": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
